Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (b09805e) 100.00% compared to head (6a74578) 100.00%.
Report is 2 commits behind head on main.

@@            Coverage Diff            @@
##              main      #194   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           12        12           
  Lines          588       595    +7     
=========================================
+ Hits           588       595    +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

For #189 are you going to implement that in here or in another PR? I see it tagged on the side

@bms63 thanks for the review!

For #189 are you going to implement that in here or in another PR? I see it tagged on the side

we used nchar(type = chars) for labels and nchar(type = bytes) for length as suggested by @siye6. I think users don't need to choose between nchar(type = bytes) and nchar(type = chars). @siye6 @bms63 do you agree?

from SAS help doc:

Variable names can be up to 8 characters, and they are stored in their original case (upper or lower).
Character variables can have lengths up to 200 bytes.
Variable labels can be up to 40 characters.

For #189 are you going to implement that in here or in another PR? I see it tagged on the side

I see that the issue #91 is also tagged but there are other elements to consider regarding the length in this issue. We could have another PR for it or do it in this one; I am not sure of the best approach, but I prefer the issue not to be closed until the following points are handled:
There are two types of length:

• Metadata length
• Data length (maximum value length)

What to do in case these lengths are different:

• Data length > metadata length: This will cause the truncation of the data. This case is handled by Haven. The length is the one from the data, and a message is given.
• Data length < metadata length: In that case, to reduce the file size, the FDA requests trimming the variable length. In this case, we could give a message so the user can update the metadata length.
• Metadata length is missing: For character variables, should we assign the data length or 200? In order to trim the variable, I suggest using the data length. Per define.xml specification, the --DTC variables should have the length missing in define.xml, so this scenario is probable.

Note:
In theory, the metadata length corresponds to the planned length and might be different from the data length. In practice, the metadata and the data length are usually similar.

The FDA requests trimming the variable across datasets. For example, if AVISIT data length = 40 in ADLB and AVISIT length = 30 in ADVS, the AVISIT length should be 40 for all datasets. In this case, the metadata length will be different from the data length. I don’t think we can check this scenario, but maybe we could mention this point in the documentation.

Shall we mention the byte/character issue in the documentaion for xportr_length() would that suffice?

@bms63 thanks for the review!

For #189 are you going to implement that in here or in another PR? I see it tagged on the side

we used nchar(type = chars) for labels and nchar(type = bytes) for length as suggested by @siye6. I think users don't need to choose between nchar(type = bytes) and nchar(type = chars). @siye6 @bms63 do you agree?

from SAS help doc:

Variable names can be up to 8 characters, and they are stored in their original case (upper or lower).
Character variables can have lengths up to 200 bytes.
Variable labels can be up to 40 characters.

Hi @kaz462 , thanks for the updates and sorry for late reply.
Regarding label, may also need to consider bytes and warning user if it is > 40, see below my test with Chinese Label

label <- "这是一段文字，用来测试在XPTversion5中作为数据集label是否会被截断"

label_len_char <- nchar(label, type = "chars")
label_len_char
label_len_bytes <- nchar(label, type = "bytes")
label_len_bytes

df <- data.frame(test = c(1, 2, 3))

haven::write_xpt(
  data = df,
  path = "test.xpt",
  version = 5,
  label = label
)

> label_len_char
[1] 40
> label_len_bytes
[1] 88

/* Use SAS convert XPT to SAS and show the label */

libname test xport "~/test.xpt";
libname sas "~";

proc copy in=test out=sas memtype=data;
	select test;
run;

proc contents data=sas.test;
run;

Label	这是一段文字，用来测试在XPTv
Encoding	utf-8 Unicode (UTF-8)

> nchar("这是一段文字，用来测试在XPTv", type="bytes")
[1] 40

Regarding label, may also need to consider bytes and warning user if it is > 40, see below my test with Chinese Label

Thanks @siye6 for providing a detailed example :) That's a good finding for haven::write_xpt - the label got truncated without any messaging, it seems haven used max 40 bytes for labels but checked against nchar(label) > 40.

For xportr, the following error message will be displayed if we use xportr_write. Do you think this message will be sufficient?

xportr_write(
    df, 
    path = "test.xpt",
    label = "这是一段文字，用来测试在XPTversion5中作为数据集label是否会被截断"
)
Error in `xportr_write()`:
! `label` cannot contain any non-ASCII, symbol or special characters.
Run `rlang::last_trace()` to see where the error occurred.

non-ASCII

Hi @kaz462 , thanks for update the warning message, regarding the non-ascii I suggest we should allow utf8 characters (including Chinese, Japanese ...etc), For eSubmission in China, one of the request is translate the foreign language data package (e.g. English) to Chinese. (variable label, dataset label, meddra, whodrug terms, primary end point related code list ..etc need to be translated from English to Chinese).

Hi @kaz462 , thanks for update the warning message, regarding the non-ascii I suggest we should allow utf8 characters (including Chinese, Japanese ...etc), For eSubmission in China, one of the request is translate the foreign language data package (e.g. English) to Chinese. (variable label, dataset label, meddra, whodrug terms, primary end point related code list ..etc need to be translated from English to Chinese).

@siye6 Thanks for the further information! I've created two issues below -

1. Dataset label truncated after write_xpt tidyverse/haven#746
2. allow utf8 characters  #202

@kaz462, @cpiraux and I met and think this is ready to merge!! yay!